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This research analyzed 12 Minnesota Technical Colleges’ Program Assessments in terms of: l.Did assessments 
of their Career and Technical education programs focus on results-related outcomes ? 2. Were program 
assessment measures “related” to programs’ goals ? 3. Were targeted programs’ goals attained? Conclusions: 
1. Report findings had limited relationships to results-related findings. 2. Assessment outcome measures tended 
not to measure proposed outcomes. 3. Reports generally claimed that they had evidence that program goals 
were attained. 
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Background 

When organizations are granted federal monies they are required to assess the effectiveness with which these monies 
are used. Typically the funding recipients conduct an assessment that measures the activities that occur rather than 
actually assessing the outcomes in terms of meaningful results. In this study a theory is proposed for what is 
essential to the assessment of organizational performance outcomes. The theory is tested using the case of the 
program performance of federally funded Career and Technical Education Programs in 12 Minnesota Technical 
Colleges. A conclusion is drawn about the merit of the theory and its application in those 12 Technical Colleges. 

Introduction to the Problem 

In a survey of recent publications about evaluation of programs in postsecondary education, the authors found few 
journal articles. Indeed, one theme was the lack of effective evaluation of programs in postsecondary education 
(Connal, 2001; Parker, Shaw, & McGuire, 2003). Reasons given were that evaluation is hard to do well (Parker et 
al), and for some institutions the perception is that doing a program evaluation is tantamount to indicating that the 
program has a problem and so is in trouble. In the authors’ experience an additional reason that is given is the time 
and cost of good evaluation seem prohibitive in a time of tight resources in the academic community. 

In the literature, the authors found two typical approaches to the evaluation of programs in postsecondary 
education. Brinkerhoff was cited by Parker, Shaw, & McGuire (2003) and Zinser (2003), Kirkpatrick was cited by 
Philips (1998) as sources that shaped or contributed to their evaluative model. Brinkerhoff suggested that evaluation 
has six levels including immediate outcomes and examined how long these outcomes last. Kirkpartick (1998) 
proposed four levels of outcomes: (a) satisfaction, (b) changes in knowledge, skills and attitudes; (c) changes in on- 
the-job behavior; and (d) changes in business-impact variables. Philips (1998) added a fifth level of return on 
investment. However, when the studies were analyzed, there was little evidence of the actual measurement of 
knowledge, of skills that were learned, nor of systemic or financial outcomes. 

There appear to be three forces pushing for evaluations of postsecondary programs. One force is the changing 
nature of the relationship between the business community and technical colleges, in which “there is an increasing 
sense of urgency to work with individual industries to offer content specific courses” (Zinser, 2003, p. 51). Another 
force is the growing pressure from federal funding agencies to require outcomes based evaluation. A third force is 
the pressure from academic accrediting organizations to require more outcomes based evaluation, including 
provision of evidence of student learning outcomes in terms of knowledge, skills and abilities (Accrediting 
Commission for Community and Junior Colleges of the Western Association of Schools and Colleges, 2003). 
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The Relevance of the Problem for HRD 

This paper involves an assessment of the evaluations of 12 Technical Colleges, looking specifically at the extent to 
which the evaluations measured meaningful results. The results indicate that typically the evaluations did not reveal 
meaningful information about outcomes. As a result the evaluations could not be used as effective tools to assist the 
colleges to improve their performance. 

There are two ways in which the evaluation of technical colleges are relevant for HRD. First, technical colleges 
are organizations, and as such are a legitimate example of the environments addressed by HRD practitioners. 
Examining the performance of the technical colleges’ evaluation system is, in itself, relevant to those who work with 
technical colleges and similar organizations. Second, if we accept Swanson and Holton's definition of HRD, HRD is 
“a process for developing and unleashing human expertise through organization development and personnel training 
and development for the purpose of improving performance” (Swanson & Holton, 2001, p. 4). One element of HRD 
is accurately assessing organizational outcomes both to discern need and as a baseline against which change will be 
measured. The evaluation of outcomes thus is a critical aspect of HRD. A look at how evaluation actually occurs in 
one system is insightful and instructional. 

Assessing Results Versus Measuring Activities 

The underlying belief that informs this study is that the assessment of organizational outcomes can (and should) 
provide valuable information for organizations. This may sound obvious, but in fact the practice of measuring 
meaningful results -oriented outcomes is less common than one might imagine. In addition, the tendency of 
organizations to avoid assessment is common knowledge. For organizations that do assess their programs, 
frequently what is studied is activities, rather than outcomes. Swanson and Holton (1999) criticized the often-used 
evaluation model of Kirkpatrick (1998) because “a key failure is its emphasis on reactions versus the fundamental 
performance results...” (5) of the organization. Therefore, the desire to overcome this tendency to focus assessment 
efforts inappropriately is the driving force behind this research effort. (Note: Educational institutions tend to use 
the term “evaluation” while business organizations typically use the term “assessment.” Both will be treated as 
being interchangeable for the purposes of this document, since as the HRD-oriented Results Assessment System is 
being applied to educational institutions) 

Weaknesses of Typical Assessment Models 

One of the commonly used approaches to evaluation is represented by the model developed by Kirkpatrick 
(1998). Swanson and Holton (1999) reported their belief that the commonly used Kirkpatrick (1998) model is 
flawed. They offered the following criticisms of the Kirkpatrick assessment model: 

1 . The model is based on assumptions that are not valid and have not been validated during its 40 years of usage. 

2. The model's conceptualized “levels” of assessment hypothesize that reactions are a proxy for learning; learning 
is a proxy for behavior; and behavior is a proxy for results. Swanson and Holton assert that none of these 
assumptions is valid. 

3. The model is not research based and is not confirmed by research. 

4. The model does not address the key evaluation and measurement issues of validity and reliability. 

Overview of the Results Assessment System. 

Swanson and Holton (1999) defined their Results Assessment System (RAS) as “The organizational 
process of determining whether meaningful and valued outcomes are achieved from human resource 
development interventions.” The RAS is organized into the three Domains of two categories each, as 
depicted below: 

Results Assessment Domain #1 : Performance Results 

System : Units of mission-related outputs in form of valued goods and services related to core 

organizational, work processes, and group or individual contributors to the organization 
Financial : The conversion of the outputs attributable to interventions, into financial outcomes 
Results Assessment Domain #2: Learning Results 

Knowledge: Mental achievement acquired through study and experience 

Expertise: Human behaviors having effective results & optimal efficiency, acquired through study & 

experience within a specialized domain 

Results Assessment Domain #3: Perception Results * 

Participant perceptions: Perceptions of people with firsthand experience with systems, processes, 

goods, and/or services. 

Stakeholder perceptions: Perceptions of leaders of systems and/or people with a vested interest in the 
desired results and the means of achieving them 
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The RAS approach leads evaluators through decisions about which domains will be assessed, and what 
kinds of data will be used in each aspect of assessment. The RAS approach was used to guide the analysis 
of the 12 selected Technical Colleges’ APRs in regard to the following: 

Measurability : The colleges explored the extent to which their evaluations truly measured results, not just the 
counting of activities, for each institution’s assessment strategies. Those rankings were also compared to the 
average rating of all 12 institutions’ Annual Performance Reports (APRs). 

Relevance : The colleges explored the extent to which assessment foci and their measurements were related to the 
proposed CTE program goals of each institution's assessment strategies, as rated by the researchers. Those rankings 
were compared to the average rating for all 12 institutions. 

Attainment : The colleges explored the extent to which their programs’ outcome measures indicated that proposed 

goals were attained/met. Each institution’s assessment strategies was rated by the researchers. Those rankings were 
compared to the average rating for all 12 institutions' APRs. 

Research Methodology 

The Case 

Minnesota’s Technical College System receives a major portion of its operational funds from the US Department of 
Education. That funding is authorized by Congress via the Carl Perkins Act. These Perkins Act funds are granted 
with the expectation that recipient institutions will orient their funded activities on 12 “Required Indictors” which 
describe required program funding categories. In addition, Perkins-funded career and technical education (CTE) 
programs must be assessed in terms of four “Core Indicators” which specify the high priorities program outcomes. 
Figure 1 depicts those Required and Core Indicators as they relate to one another, as well as the major Results 
Assessment Categories addressed by Swanson and Holton’s (1999). Results Assessment System. 

Research Questions 

The researchers have noted that Minnesota’s Technical Colleges often seem to focus on assessing their CTE 
programs’ activities rather than on actual performance-related results. In addition, the assessment measures used 
seem to tend to be poorly related to program objectives. In other words, these Technical Colleges’ program 
objectives sometimes seem to be focused on outcomes of limited value for assessing programs’ outcomes and the 
measures used to assess the extent to which CTE objectives have been achieved often seem to have inadequate 
relationships to those objectives. 

Figure 1. Proposed Assessment Model for Federally-Funded Technical College Programs 



Twleve Required Indictors: Funding Priorities 

Indicator 1: Integration of academic and technical 
education 

Indicator 2: Experience in and understanding of all 
aspects of industry 

Indicator 3: Technology in vocational-technical 
education. 

Indicator 4: Professional Development 

Indicator 5: Evaluation of vocational-technical 
education programs 

Indicator 6: Continuous improvement for vocational- 
technical education programs 



Indicator 7: Effectiveness of services and activities 

Indicator 8: Broad-based community involvement 

Indicator 9: Special population learner 

accommodation! s) and support services 

Indicator 10: Full participation of special population 
learners 

Indicator 11: Preparation for nontraditional training 
and employment 

Indicator 12: Collaboration 
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The four core indicators and outcome priorities: 

1. Student attainment of challenging state established academic, vocational and technical skill proficiencies. 

2. Student attainment of secondary school diploma or its recognized equivalent or a proficiency credential in 
conjunction with a secondary school diploma or a post-secondary degree or credential. 

3. Placement in, retention in, and completion of post-secondary education or advanced training, placement in 
military service, or placement or retention in employment. 

4. Student participation in and completion of vocational and technical education programs that lead to 
nontraditional training and employment. 

This research was designed to analyze selected Minnesota Technical Colleges’ APRs and the extent to which 
they are/are not “results oriented.” This research is designed to provide a initial baseline that, hopefully, will 
encourage subsequent research that will enhance the utility and effectiveness of CTE program assessment strategies. 
Therefore, the researchers examined the APRs of 12 selected Minnesota Technical Colleges in an effort to answer 
the following research questions: 

1. To what extent do repotted assessment goals and targeted assessment measures focus on “results,” as related to 

the Swanson's (1999) Results Assessment Model? 

2. Are the measures contained in the APRs “relevant” to the proposed goals? 

3. To what extent do the reported assessment measures indicate that the proposed goals were attained? 

The researchers reviewed and analyzed each component of the 12 APRs. A Likert-type scale was used to rate 
these APRs in relation to each of the above key examples of those comments are presented below for selected 
Technical Colleges. 

The process for assessing each APR used the following rating form depicted in Figure 2 below: 

Figure 2. Rating Scale Applied To Each Technical College’s Assessment Processes 



Perkins Indicator #: 








RAS Categories 


Measurability of 


Outcome Measure’s 


Extent to Which 


Represented: 


Proposed Activities? 


Relevance to Proposed 


Measures Indicate 




(e.g., RESULTS- 


Activities? 


Objectives Were 


• System 

• Financial 

• Knowledge 


Oriented vs. Activity 
count) 




Attained? 


• Expertise 


2= Good 


2= Good 


2= Good 


• Participant Perception 


1= Needs Improvement 


1= Needs Improvement 


1= Needs Improvement 


• Stakeholder Perception 


0= None Observed 


0= None Observed 


0= None Observed 



RAS Category(ies) : 

Average Rating: 



Measurability 



Relevance 



Attainment 



After reviewing each of the 12 APRs, a series of insights were developed in relation to each 
of the three research questions. Those insights for this case study are listed below: 

Research Question 1. To what extent do reported assessment goals and targeted assessment measures focus on 
“results,” as related to the Swanson’s (1999) Results Assessment Model? 

Accomplishment statements often differed substantially from targeted results measures. For instance, a “2% 
increase in Annual Enrollment Rate” was reported versus accomplishments expressed in terms of % of total 
population enrolled. The growth in numbers reveals nothing of the quality of the students, nor of their ability to 
perform in their studies. Many assessment strategy statements seem unrelated to targeted measures. Actual measures 
of knowledge and related expertise levels/gains are seldom addressed. Three examples of this trend follow: 
a. Many professional development and student learning measures reported by participating institutions were focused 
on numbers of students or events: 

Number of workshops conducted 
Number of workshop participants 
Number of internships 



49-2 





1136 



Number of training sessions 
Number of Continuing Education Units 
Number of special populations enrolled. 

And were not focused on performance related outcome such as: 

Gain scores 
Skill standards attained 
Number graduated 
Post-training success 

b. New staff position implementation efforts were assessed in terms of numbers of persons: 

Number of new staff hired 
Percentage appointed 

And were not focused on performance related outcomes such as: 

Staff members’ achievements in terms of job-related accomplishments 

c. Career Fairs and Community Tours were assessed in terms of numbers of events or attendance: 

Number of fairs or tours 

Number of students, parents, business 

Participation level (attendance) 

And were not focused on performance related outcomes such as: 

Perceptions of volume (students and businesses) 

Number of subsequent CTE enrollees 
Increase in business input/on-going collaboration 

Many targets focus on revising, designing for implementation infrastructure. 

Research Question 2. Were the assessment measures contained in the report relevant to the 
proposed goals? 

Student Surveys were used in numerous cases. However, student survey findings were sometimes extrapolated 
inappropriately to other time periods, to other groups, or to indicate pre-post gains in knowledge and performance. 
Enrollment changes were often used as targeted goals. However, this seemed to represent a minimally useful 
measure, because a wide variety of external factors can also influence significantly influence enrollment trends 
beyond your control (e. g., effects of changes in the economy). Participating institutions had not yet begun to focus 
on what new infrastructures will accomplish. Rather they tended to continue to focus only on counting the number 
of newly developed/revised programs. 

In each of these illustrations, the technical colleges either tended to misuse data, set goals for which attainment 
was beyond the ability of the colleges, or continued in habitual patterns of counting the number of changes (e.g., 
new programs) without looking at the quality or infrastructure needed to sustain the changes. 

Research Question 3. To what extent do the reported assessment measures indicate that the proposed goals 
were attained? Conclusions regarding the accommodation of Special Populations reported: 

All students received services. HOWEVER, the actual knowledge and expertise benefits for those students were 
not identified. All of the students who requested accommodations were reported as having been serviced. However, 
there was no indication of the effectiveness of these services, nor was there any comment about special populations 
who did not request service but who needed those services? The number of Interagency Articulation agreements 
were listed as desirable outcomes. For example, one college reported: “25% of programs will implement 

agreements.” There was no report if the agreements were used, nor if any benefits were realized, nor if there were 
any retention or transfer gains. Reports did identify several confounding influences which detracted from the value 
of the assessments. Some goals were not attained because of delays in needed data. Some goals were delayed due to 
decisions made by college management that placed other initiatives as higher priorities. Some APR's did not 
differentiate between proposal and actual budget expenditures. 

Conclusions 

1. To what extent do reported assessment goals and targeted assessment measures focus on “results,” as related to 
the Swanson's (1999) Results Assessment Model? There is a limited relationship between the Technical 
Colleges’ assessment procedures and findings for their Perkins-Funded programs and those that would have 
been suggested by the Results Assessment Model. 

2. Are the measures contained in the report “relevant” to the proposed goals? Many outcome measures did not 
meaningfully measure outcomes proposed. 

3. To what extent do the reported assessment measures indicate that the proposed goals were attained? 
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Existing measures were generally claimed as evidence that program goals were successfully attained. When 
goals were not achieved, results were limited by the following reasons: (a) Some goals were not attained 
because of delays in needed data, (b) the attainment of some goals was delayed due to decisions made by 
management that placed other initiatives as higher priorities, (c) some APR’s don't differentiate between 
proposal and actual budget expenditures. This report focuses on actual/revised budget data, if it is reported in 
the APRs. 

4. It is apparent that a basic design conflict exits in the federal model that requires 12 funding priorities that are 
conceptually disconnected from the four outcome indicators. That disconnect inhibits funded institutions from 
effectively assessing their proposed outcomes. A proposed new model is given in figure 1 . 

5. The theory being explored was supported within the 12 Minnesota Technical Colleges. To be able to generalize 
beyond this population, further studies are needed. Participating institutions should be commended for their 
participation in this effort to improve program assessment processes. The feedback provided in this report is 
intended to provide guidance for subsequent efforts, NOT to reprimand institutions for limitations in their 
current practices. 

Recommendations 

It is recommended that the 12 institutions whose APRs were reviewed in this document closely examine the extent 
to which they tended to measure activities, instead of the “results” of those activities. This will require an adjustment 
in thinking and behaving. However, it will enable these institutions’ staff to then adjust their next round of results 
measures and the nature of their strategies focused on BOTH the Perkins Indicators and the four core competencies. 
In addition, it is recommended that these institutions begin to expand the focus of their results measures beyond 
issues related to “system” assessment results categories. The assessment categories of “knowledge” and “expertise” 
are recommended for immediate attention. The “financial” assessment is strongly recommended for implementation 
within the next three years. The financial category may represent the most valuable area for attention. However, a 
period of three years is suggested for the development implementation efforts in this assessment category because it 
will require the greatest growth in measurement sophistication and the most effort to establish appropriate data 
collection and analysis procedures. Finally, the existence of the confusing, potentially incompatible, priorities 
mandated within the funding policies of the Perkins Act should be acknowledged and considered. Assuming that 
this joint set of priorities continues, grant recipients are encouraged to integrate their program assessment plans and 
activities into a broader perspective focused on meaningful results, not just activities. Remember, a flurry of 
activities is not necessarily the equivalent of high productivity. 
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